What is claimed is: 

L A method of designing an interactive system, the method comprising: 

providing one or more sensors, each sensor adapted to capture one or more real-time 

5 events; 

providing one or more perception features, each perception feature adapted to receive 
information concerning one or more captured events from one or more sensors and to 
generate one or more perceptions; 

providing one or more decision features, each decision feature adapted to receive one 
10 or more perceptions from one or more perception features and to generate one or more 
decisions; 

providing one or more output mechanisms, each output mechanism receiving one or 
yjl more decisions from one or more decision features, and executing one or moi-e actions; 

assigning a priority level to each perception feature and each decision feature on the 
13 basis of a plurality of factors comprising duration of an action, complexity of an action, and 
fll pre-planning requirements of an action; and 

. utilizing said priority levels to load-balance the running of perception features and 

decision features under time constraints and normal system operation, and to select among 
conflicting actions requested by two or more decision features. 

M 

^" 2. The method of claim 1, wherein one or more of said sensors is a visual sensor, 
3. The method of claim 1, wherein one or more of said sensors is a spatial sensor. 
25 4. The method of claim 1, wherein one or more of said sensors is an auditory sensor. 

5. The method of claim 1, wherein one or more of said sensors is a tactile sensor. 

6. The method of claim 1, wherein one or more of said sensors is a balance sensor. 

30 

7. The method of claim 1, wherein one or more of said actions comprises generating speech. 
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8. The method of claim 1, wherein one or more of said actions comprises generating text. 

9. The method of claim 1, wherein one or more of said actions comprises generating 
phonemes. 

10. The method of claim 1, wherein one or more of said actions comprises generating an 
intonation. 

11. The method of claim 1, wherein one or more of said actions comprises generating an 
animated movement. 

12. The method of claim 11, wherein the movement includes facial expression, 

13. The method of claim 1 1, wherein the movement includes body movement. 

14. The method of claim 11, wherein the movement includes hand gesture. 

15. The method of claim 11, wherein the movement includes arm gesture. 

16. The method of claim 1 1, wherein the movement includes visemes, 

17. The method of claim 1, wherein one or more of said actions comprises generating 
physical movement. 

18. The method of claim 17, wherein the movement includes facial expression. 

19. The method of claim 17, wherein the movement includes body movement. 

20. The method of claim 17, wherein the movement includes hand gesture. 

21. The method of claim 17, wherein the movement includes arm gesture. 
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22. The method of claim 17, wherein the movement includes visemes. 

23. The method of claim 1, further comprising: 

selecting one of said decision features; and 

assigning a high priority level to the selected decision feature on the basis of one or 
more criteria, including: 

a. when output from the selected decision feature results in an action requiring less 
than approximately 500 milliseconds to perform; 

b. when output from the selected decision feature results in a simple action that 
requires few motors to perform; 

c. when the input to the said decision feature are output produced by a perception 
feature or features with a high priority; 

d. when the inputs to the decision feature come from a relatively few perception 
features; 

e. when the decision feature requires relatively simple processing; 

f . when the amount of preplanning required for the decision is low or none. 

24. The method of claim 1, further comprising: 

selecting one of said decision features; and 

assigning a medium priority level to the selected decision feature on the basis of one 
or more criteria, including: 

a. when the selected decision feature results in an action requiring approximately 
500-2000 milliseconds to perform; 

b. when the selected decision feature receives output from a relatively large number of 
features; 

c. when the selected decision feature is clearly related to hand-holding of the 
interaction between the system and a system user; 

d. when the input to the said selected decision feature is from a perception feature 
with a high or medium priority level; 

e. when the amount of preplanning required for the decision is intermediate; 
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f . when the action resulting from the decision feature's decision output is complex, or 
requires a high number of motors to be performed. 

25. The method of claim 1, further comprising: 

selecting one of said decision features; and 

assigning a low priority level to the selected decision feature on the basis of one or 
more criteria, including: 

a. when the duration of an action resulting from said selected decision feature is 
approximately longer than 2000 milliseconds; 

b. when said selected decision feature requires complex processing; 

c. when said selected decision feature requires large amounts of pre-planning; 

d. when said selected decision feature is related to a knowledge-based topic other than 
the interaction itself; 

e. when the input to said selected decision feature is from a feature with a medium or 
low priority level. 

26. The method of claim 1, further comprising: 

selecting one of said perception features; and 

assigning a high priority level to the selected perception feature on the basis of one or 
more criteria, including: 

a. when input of said selected perception feature is a single data entry, or a collection 

of very few data entries; 

b. when input of said selected perception feature comes from a single sensor; 

c. when the processing time of said selected perception feature supports one or more 
actions having a perception-to-action loop time that is shorter than 500 milliseconds; 

d. if one or more decision features supported by said selected perception feature 
supports one or more outputs that result in an action requiring less than approximately 500 
milliseconds to perform; 

e. when data needed already exists as raw data; 

f . when data gathering takes roughly 300 milliseconds or less; 
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when input of said selected perception feature is produced by one or more 
perception features each of which have been assigned a high priority. 

27. The method of claim 1, further comprising: 

selecting one of said perception features; and 

assigning a medium priority level to the selected perception feature on the basis of one 
or more criteria, including: 

a. when data gathering of said selected perception feature spans approximately 
between 300 and 1500 milliseconds; 

b. when input to said selected perception feature comprises a relatively large number 
of data points from more than one sensor; 

c. when a decision feature supported by the output of said selected perception feature 
is directly related to an interaction between said interactive system and its users; 

d. when data processing of said selected perception feature requires information from 
various parts of said interactive system, including the context of an interaction; 

e. when said selected perception feature supports perception-to-action loop time 
between 500 and 1500 milliseconds; 

f. when said selected perception feature takes between 300 and 1500 milliseconds to 
produce output; 

g. when said selected perception feature's input is the output of other perception 
features that have been assigned a medium or high priority level; 

h. when said selected perception feature's output supports a decision feature whose 
resulting action takes between 500 and 1500 milliseconds to perform. 

28. The method of claim 1, further comprising: 

selecting one of said perception features; and 

assigning a low priority level to the selected perception feature based on one or more 
criteria, including: 

a. when data gathering of said selected perception feature requires longer than 
approximately 1500 milliseconds; 
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b. when data processing of said selected perception feature requires complex data or 
knowledge structures for producing output; 

c. when said selected perception feature produces complex, knowledge-based output; 

d. when said selected perception feature's output supports processes for high-level 
decision making such as complex natural language processing or planning ahead; 

e. when data processing of said selected perception feature takes more than 1500 
milliseconds to perform; 

f. when data processing of said selected perception feature requires information from 
multiple parts of said interactive system, including the context of an interaction; 

g. when said selected perception feature's input is the output of other perception 
features that have been assigned a low or medium priority level; 

h. when said selected perception feature's output supports a decision feature whose 
resulting action takes longer than 1500 milliseconds to perform; 

i. when said perception feature is difficult or impossible to decompose into small 
perceptual features; 

j. when said perception feature typically cannot support a perception-to-action loop of 
shorter than 1500 millisecond duration. 

29. The method of claim 1, wherein the method is used to design an interactive system that 
implements one or more anytime algorithms. 

30. The method of claim 1, wherein the method is used to design an interactive system that is 
implemented in a distributed, asynchronous architecture. 
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